Extending Coverage Criteria by Evaluating Their Robustness to Code Structure Changes

Code coverage is usually used as a measurement of testing quality and as adequacy criterion. Unfortunately, code coverage is very sensitive to modi cations of the code structure, and, therefore, the same test suite can achieve di erent degrees of coverage on the same program written in two syntactically di erent ways. For this reason, code coverage can provide the tester with misleading information. In order to understand how a testing criterion is a ected by code structure modi cations, we introduce a way to measure the sensitivity of coverage to code changes. We formalize the modi cations of the code structure using semantic preserving code-to-code transformations and we propose a framework to evaluate coverage robustness to these transformations, extending actual existing coverage criteria. This allows us to de ne which programs and which test suites can be considered robust with respect to a certain set of transformations. We can identify when the obtained coverage is fragile and we extend the concept of coverage criterion by introducing an index that measures the fragility of the coverage of a given test suite. We show how to compute the fragility index and we evidence that also well-written industrial code and realistic test suites can be fragile. Moreover, we suggest how to deal with this kind of testing fragility.